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Abstract 

This report summarizes our research findings on the Black box testing of Runway 
Incursion Advisory Alerting System (RIAAS) and Runway Safety Monitor (RSM) 
system. Developing automated testing software for such systems has been a problem 
because of the extensive information that has to be processed. Customized software 
solutions have been proposed. However, they are time consuming to develop. Here, we 
present a less expensive, and a more general test platform that is capable of performing 
complete black box testing. The technique is based on the classification of the anomalies 
that arise during Monte Carlo s imulatio ns. In addition, we also discuss a generalized 
testing tool {prototype) that we have developed. 

1. Introduction 

In order to get insight into the techniques for black box testing, we have adopted Runway 
Incursion Advisory Alerting System (RIAAS) developed by Rannoch Corporation and 
Runway Safety Monitor (RSM) system developed by NASA LaRC. Clearly, due to 
privacy and patent reasons, we were not given access to the source. The only thing that 
was available was executable code. So our task was to evaluate these two pieces of 
software and learn the effectiveness of our tools. 

The simulator to test the systems was developed by Rannoch Corporation. It was 
primarily developed to test RIAAS software. However, we used the simulator to test the 
RSM simulator also. A schematic diagram of die simulator is shown in Figure 1. 



Figure 1. Schematic Diagram of the Test Architecture 
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To start the simulator, the user chooses the setup parameters. These include die airport 
and runway type, the orientarion/position of the ownship (aircraft undo: study) and that of 
the traffic (the other aircraft), single or Monte Carlo simulation, etc. Once the parameters 
are set, the simulator starts running. During the execution, it has access to the airport data 
of the chosen airport. 

The simulator generates an output containing the information with details of ownship and 
traffic. The details include (for every 0.5 seconds) the position information, whether or 
not an alert generated, if so when it was generated, etc. In a Monte Carlo situation, the 
one that we focused on, several runs (in our case 10,000 runs) are run for each setup. 
Thus, the details file contains details of the aircrafts for each of these runs at each 0.5 sec 
duration. The summary file contains a summary for each run. Details for each run include 
the positions of the two aircrafts and the time at which simulation started, whether or not 
an alertftvarning was generated, and whether or not a collision (or near-coiiision) 
occurred during the run. In addition, at the end, it prints the number of collisions and 
near-collisions in the entire simulation (i.e., 10,000 runs). 

In order to compare the performance of RIAAS and RSM, we have run both RIAAS and 
RSM with simulator input In order to do further analysis on the ownship and traffic data, 
we have developed an analyzer program. The program looks at the summary output file, 
and categorizes each run into one of several classes as described in the next section. The 
results from the analyzer are summarized in this report. In consultations with NASA, we 
have identified 24 scenarios that need to be tested. In addition, several other s imulati ons 
were carried out to test the two software packages. During the testing, we identified 
several anomalies with the behavior of the simulator. While these are not reflective of the 
errors in the RIAAS and RSM software, it did cause some problems in our testing and in 
distinguishing between simulator errors versus the test software errors. 

In this report, we first describe a classification of the simulation runs. We then describe 
the results of our analysis of the selected scenarios. Each scenario has been tested with 
10,000 runs. 

2. Classification of Monte Carlo Runs 

The first step of a black box testing software is to classify the inputs into different types. 
In the system under consideration (RIAAS/RSM), where input is generated randomly 
using Monte Carlo generator, we classified inputs into three categories. Figure 2 shows 
our classification. It is also schematically shown in Table 1. 

Collisions: An event is defined as collision if the separation distance between ownship 
and traffic is less than 200 ft. 

Near-collision: An event is defined as near-collision if the separation distance between 
ownship and traffic is within 200-300 ft. 

Safe or No-collision: An event is defined as safe if the separation distance between 
ownship and traffic is greater than 300 ft. 
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Ideally, the software under test (SUT) should be able to correctly identify each case (i.e M 
scenario). However, ineffective software may categorize collisions as non-collisions 
(namely, missed alerts) and non-collisions as collisions, (namely, false alerts). We 
measure die effectiveness of the SUT (RIAAS/RSM) based on the classification in Figure 
2. For example, die most effective software would have zero percent false alerts , and 
zero percent of avoid possible cases. The avoid impossible cases are those where it is 
impossible to alert collision (or near-collision) by any such software, undo: the given 
assumptions. 



Figure 2. Classification of Monte Carlo Scenario 


Category 

Result of run 

Action 

Class I 

No collisions or near collisions 

Alerts not generated 

Class II 

Collisions or near collisions 

EBB ? mSSSSSmEMMA 

Class HI 


Alerts generated 

Class IV 

Collisions or near collisions 

Alois generated 

Class Ilia 

No collisions or near collisions 

False alerts generated (there was never 
a danger of collision) 

Class mb 

No collisions or near collisions 

True alerts generated (danger was 
averted due to alerts) 

Class IVa 

Collisions or near collisions 

Alerts were correctly generated and 
collision could not have been avoided 

Class IVb 

Collisions or near collisions 

Alerts were correctly generated, and 
collision could have been avoided 


Table 1. Classification erf Monte Carlo Runs 
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3. Analysis of the Software Under Test (SUT) 


In this section, we describe the way we tested the black box software (RIAAS and RSM), 
and die findin gs from the analysis. 

3.1 Generation of input. 

One of the primary considerations in conducting blackbox testing is choosing the right 
type of test cases that can ensure coverage of all aspects of the software under test (i.e., 
RIAAS and RSM). Due to the nature of the software and the environment in which it is 
supposed to run, we have resorted to domain expert knowledge. In particular, we have 
chosen the scenarios summarized in Tables 2 and 3. Table 2 describes scenarios used for 
Crossing runway situation and Table 3 for Single runway situation. For each scenario in 
these tables, 10,000 random input cases were generated and the results categorized into 
Classes I-IY as described above. Each input consists of starting the ownship and the 
traffic at certain positions, either in taxiing, arrival, or departure operations. The position 
of the aircrafts is supplied to the software at frequent intervals, until the designated time 
elapses. This si repeated for each run. 

3.2 Output Analysis 

The SUTs (RIAAS and RSM), when subjected to different input scenarios generated by 
the data generator, make decisions as to whether or not to generate an alert, and when to 
generate it During the run, the simulator (same as the data generator) keeps track of the 
position of the aircrafts as well as the action taken by the software. It also prepares 
statistics such as the number of collisions, near collisions, missed detections and other 
supplementary information. This information is analyzed by an analysis program that we 
developed. The analyzer checks whether or not the software’s behavior is as expected. 
The program also classifies each experiment into one of the types that we described in 
Figure 2 and Table 1. For example, a near collision event is identified if the closest 
distance between the ownship and traffic is between 200 and 300ft; the alert flag 
information is used to identify whether or not alerts were generated. Upon classification 
of the samples, the analyzer statistics are independently evaluated and compared with the 
statistics given by the simulator output to check for inconsistencies. 


4. Detected anomalies 

During our analysis of the software, we identified several anomalies. Some of them are 
manifestations of the simulator (developed by Rannoch corporation) and others may be 
limitations of the software (RIAAS and RSM). Below, we demonstrate the application of 
testing program with some examples. 

4.1 Anomaly 1: Collisions and Near Collisions occur without alerts. 

This is a serious problem and a weakness of the software. For e xamp le: 

Consider a near collision case of RIAAS in a single runway scenario where the Owship 
state is in arrival and traffic state is in taxi. Assuming the type of evasive action taken on 
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Ownship is RCA and none for traffic, we have the following output for the simulator 
(verbatim). 


Note: The description of examples given below is as follows. 

Description 

Example 

Summary 

201 

Run number 

15.0 


0 

Traffic start time ] 

984.3 

Distance from traffic to threshold 

8202.5 

Distance from ownship to threshold 

0 

The time delay at which piiot receiving RTA 

0.0 

The time delay at which pilot receiving RCA 

0.0 

The time at which both ownship and traffic are 3d closest 

55.0 

The distance at which both ownship and traffic are 3d closest 

262 

The altitude which boh ownship and traffic are 3d closest 

0 

The time at which both ownship and traffic are 2d closest 

55.0 

The distance at which both ownship and traffic are 2d closest 

262 

The altitude which both ownship and traffic are 2d closest 

0 

Evasive action taken for both ownship and traffic 

N/A 

Ownship evasive maneuver 

None 

The speed of ownship at which evasive action is taken 

0.0 

The speed of traffic at which evasive action is taken 

0.0 

Distance from ownship to threshold 

0.0 

Speed of ownship at which ownship and traffic both are 3d closest 

135.0 

Speed of traffic at which ownship and traffic both are 3d closest 

16.6 

Speed of ownship at which both ownship and traffic are 2d closest 

135.0 

Speed of traffic at which both ownship and traffic are 2d closest 

16.6 

Ownship x position at which berth ownship and traffic are 3d closest 

0.0 

Ownship y position at which both ownship and traffic are 3d closest 

1026.1 

Traffic x position at which both ownship and traffic are 3d closest 

258.3 

Traffic y position at which both ownship and traffic are 3d closest 

983.8 

RTA number 

None 

RCA number 

None 

Time at which traffic has taken evasive action 

N/A 

Traffic evasive maneuver 

None 


Table 2. Explanation for the Summary Output 


Summary 201 15.0 0 984.3 8202.5 0 0.0 0.0 55.0 262 

0 55.0 262 0 N/A None 0.0 0.0 0.0 135.0 16.6 

135.0 16.6 0.0 1026.1258.3 983.8 None None N/A None 

Note: This is one of the cases (case # 9) of Table 2 listed below. 

In the above example, we have the start time for ownship is 15s and start time for traffic 
is 0s. The separation distance between ownship and traffic is 262ft and altitude being Oft. 
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Clearly this is a near collision case, and alerts woe not issued thus falls into class II 
category 


Collision: 

Consider a collision case qfRIAAS in a single runway scenario where the Owship state is 
in arrival and traffic state is in taxi. Assuming the type of evasive action taken on 
Ownship is none and RCA for traffic, number cf missed detection being 338. 

Summary 339 23.0 0 656.2 8202 JO 0.0 0.0 38.5 52ft 

15ft 38.5 50 15 N/A None 0.0 0.0 0.0 135.0 20.0 

135.0 20.0 0.0 684.3 41.1 655.7 None None N/A None 

In die above example, we have the start time for ownship is 23s and start time for traffic 
is 0s. The separation distance between ownship and traffic is 52ft and altitude being 15ft. 
clearly this is a collision case, and alerts were not issued thus falls into class n category 

4.2 Anomaly 2: All 10000 Iterations are not generated for some of the scenarios. For 
these cases no collision information is printed in die simulator output file. 

Consider a collision case of RIAAS in a single runway scenario where the Owship state is 
in arrival and traffic state is in taxi. Assuming the type of evasive action taken on 
Ownship is RCA and RCA for traffic, with random generator ON for ownship and OFF 
for traffic. 

Summary 8989 24.0 0.0 1804.6 8202.5 0 0.0 0.0 66.5 1734 

0 66.5 1734 0 N/A None 0.0 0.0 0.0 122.9 20.0 122.9 

20.0 0.0 1570.1-1718.2 1804.1 None None N/A None 

Note: This is one of the cases (case # 12) of Table 2 listed below. 

In the above example, in the simulator output file summary, 8989 is the last run that got 
generated and stopped after thaL(Complete 10000 iterations are not generated) and no 
collision count:0 is printed. 

43 Anomaly3: Mismatch in the number erf collisions and the number erf near 
collisions printed in the simulator output file and scanned output 


Consider RSM, single runway scenario where the Owship state is in taxi and traffic state 
is in arrival Assuming the type of evasive action taken on Ownship is none and none for 
traffic, with random generator ON for ownship and OFF for traffic. 


Metric 

Simulator output 

Analyzer emtput 

Number erf collisions 

1150 

1281 

Number erf near-ex>llisions 

790 



Note: This is one of the cases (case # 2) of Table 3. 
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In the above example, in the simulator output file, the number of collisions occurring 
woe 1150, but the actual number of collisions (as reported by the analyzer) woe 1281, 

44 Anomaly 4: In the simulator output file run numbers are being generated 
sequentially till a certain valne, with a break in the middle and continuing again 
sequentially from there onwards. 

Consider a case af RIAAS in a single runway scenario where the Owship state is in 
arrival and traffic state is in taxi Assuming the type of evasive action taken on Ownship 
is RCA and RCA for traffic, with random generator ON for ownship and OFF for traffic. 

Summary 6084 44.0 0.0 984.3 8202.5 0 0.0 0.0 82.5 2290 

15 82.5 2290 15 N/A None 0.0 0.0 0.0 135.0 20.0 135.0 

20.0 0.0 684.3 -2270.1 983.8 None None N/A None 

Summary 6115 34.0 0.0 3281.0 8202.5 0 0.00.0 83.0 2349 

0 83.0 2349 0 N/A None 0.0 0.0 0.0 91.3 20.0 91.3 

20.0 0.0 2744.6-2287.4 3280.5 None None N/A None 

Note: This is one of the cases (case # 12) of Table 2 listed below. 

In the above example, in the simulator output file, the run numbers were generated from 

1 to 6084, the run numbers from 6084 to 6114 were missing, and from 6115 onwards the 
run numbers are generated again. 

4.5 Anomaly 5: In the simulator output file some Run numbers are being repeated 
and the total number of run numbers generated is greater than 10000. 

Consider a case of RIAAS in a single runway scenario where the Owship state is in 
arrival and traffic state is in taxi Assuming the type of evasive action taken on Ownship 
is RTA/RCA and RCA for traffic, with random generator ON for ownship and OFF for 
traffic. 


Summary 

9 

45.0 

0.0 3445.1 8202.5 0 

0.0 

0.0 

94.5 

2756 

0 94.5 

2756 

0 

N/A None 0.0 0.0 

0.0 

88.8 

20.0 

88.8 

20.0 

0.0 

2820.6 

-2684.1 3444.6 None 

None 

N/A 

None 


Summary 

1 

17.0 

0.0 4101.3 8202.50 

0.0 

0.0 

71.5 

1994 

0 

71.5 

1994 

0 N/A None 0.0 

0.0 

0.0 

64.5 

20.0 

64.5 

20.0 

0.0 

3467.8-1890.7 4100.8 None 

None 

N/A 

None 


In the above example, in the simulator output file, the first run number generated is 9 and 
again started from 1. The run number 9 is generated again thereafter. This is continued 
for some more run numbers. 

4.6 Anomaly 6: The separation distance between the traffic and ownship is very 
large and the time at which they are closest is much greater and alert is given for 
one ran. However, when their separation distance Is small and the time at which 
they are closest is smaller, alert is generated for another ran. 
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Consider a case of RIAAS in a single runway scenario where the Owship state is in 
arrival and traffic state is in taxi. Assuming the type of evasive action taken on Ownship 
is RCA and RCA for traffic, with random generator ON for ownship and OFF for traffic. 


Example run: 


Summary 

10 

14.0 

51.0 

820.3 8202.50 

0.0 

0.0 

53.0 

383 

9 53.0 

383 

9 

N/A 

None 0.0 0.0 

0.0 

135.0 

4.4 

135.0 

4.4 0.0 

798.2 

382.4 

819.8 

None None N/A 

None 




Summary 

1 

17.0 

22.0 

3117.0 8202.50 

0.0 

0.0 

63.5 

2403 

2161 65.5 

937 

2260 27.0 

Climb 135.0 8.7 

0.0 

135.0 

20.0 

135.0 

20.0 

0.0 

2507.2 

-855.8 3116.5 1 None 

N/A 

None 



Alert 1 

RTA 

26.5 

0 

-5924 362 355 

3116 

0 

28,5 

0 

-5468 

432 

319 

3116 

0 






Note: This is one of the cases (case # 12) of Table 2 listed below. 

The separation distance in the 10* run is 383ft, which is very less and expects an alert 
But the separation distance in the l rt run is 2403ft for which alert is given. 

4.7 Anomaly 7: Alert is generated after the time at which both ownship and traffic 
are closest 

Consider a case of RSM in a crossing runway scenario where the Owship state is in 
departure and traffic statte is in departure. Assuming the type cf evasive atction taken on 
Ownship is none and none for traffic, with random generator OFF for ownship and ON 
for traffic. 

Summary 6805 0.0 18.0 0.0 656.2 0 0.0 0.0 13.5 3589 

0 13.5 3589 0 N/A None 0.0 0.0 656.2 68.0 191.0 68.0 

191.0 3674.412646.1 6227.315168.1 None 1 N/A None 

Alert 6805 RCA 2R5 5545 10776 0 6227 14672 0 35.5 6806 

9514 183 6227 13839 0 

Note: This is one of the cases (case # 19) of Table 1 listed below. 

In the above example die time at which both ownship and traffic are closest is 13.5s, but 
alert is generated at 2&5s. 
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As mentioned in the introduction, in consultations with NASA, we have run Monte Carlo 
runs on 24 scenarios. This sample represents typical scenarios of interest for collision 
detection and avoidance software. The scenarios for crossing runways are summarized in 
Table 3. The ones for single runway are summarized in Table 4. 

Table S summarizes the output of RIAAS and the analysis output in each of the 24 
scenarios for single runway (Table 4). For each case, the results from Rarmoch’s 
simulator as well as the output from our analysis program are included here. For example, 
in case 22, the simulator identified 376 collisions and 111 near-collisions. These are out 
of 10,000 Monte Carlo runs run with the same scenario (case 22). Further categorization 
of these cases is available through the analysis program output. Here, it shows an analysis 
of all 10,000 runs. 5805 resulted in no collisions and there were no alerts. This is a 
normal situation and classified under Class I. Then, 3059 resulted in no collisions but 
there was an alert. Some of the alerts may have been avoided due to the alerts generated 
by RIAAS while some may have been false aims. But this table does not distin guish 
between these two cases. (The only way to separate these cases is to make an identical 
runs with and without invoking the alerts. But this was not possible due to Monte Carlo 
simulation with randomized inputs.) 

Now let us Consider the runs that resulted in collision or near-collision. There are 
(10,000-5805-3059) or 1136 of these. Of these, 480 (4.8% of all runs and 42.25% of all 
collisions/near collisions) wore due to collisions without alerts. These are die cases that 
need further investigation to know why alerts were not generated. 540 (5.4% of all runs 
and 47.53% of all collisions/near-collisions) were due to collisions with alerts. Here, one 
needs to investigate whether alerts should have been much earlier than when they were 
by RIAAS. Of the remaining, 20 resulted in near collisions without alerts (0.2% of all 
runs and 1.76% of all collisions/near collisions). These need further investigation. The 
rest 96 (0.96% of all runs and 8.45% of all collisions/near collisions) were due to near- 
collisions with alerts. From here, the value of the analysis program and how it provides 
more useful information than the simulator should be evident 

Tables 6, 7, and 8 are similar to this one but deal with RSM and RIAAS and single and 
crossing runways. 

Tables 9-16, also provided by our analysis program, further analyzes the cases of 
collisions and near-collisions. It gives a histogram of times between the start time of 
ownship and the time at which the traffic and ownship are the closest. So if this time is 
large enough, we can conclude that the collision could have been avoided. However, if 
this is small, we can conclude that the collision could not have been avoided. 
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Scenario# 

PBfB 

Traffic 

Evasive 
Action of 
OwnsNp 

Evasive Action 
of Traffic 

Random Generator 






Ownship 

Traffic 

1 

Arrival 

Arrival 

None 

None 

OFF 

ON 

2 

Arrival 

Arrival 

None 

None 

ON 

OFF 

3 

Arrival 

Arrival 

RCA 

None 

OFF 

ON 

4 

Arrival 

Arrival 

RCA 

None 

ON 

OFF 

5 

Arrival 

Arrival 

RCA 

RCA 

OFF 

ON 

6 

Arrival 

Arrival 

RCA 

RCA 

ON 

OFF 

7 

Arrival 

Departure 

None 

None 

OFF 

ON 

8 

Arrival 

Departure 

None 

None 

ON 

OFF 

9 

Arrival 

Departure 

RCA 

None 

OFF 

ON 

10 

Arrival 

Honariitra 

OCA 

« IV# * 

KIama 

iiwnv 

AM 

VIS 

ACT 

wrr 

11 

Arrival 

Departure 

RCA 

RCA 

OFF 

ON 

12 

Arrival 

Departure 

RCA 

RCA 

ON 

OFF 

13 


Arrival 

None 

None 

OFF 

ON 

14 

■gBSBilga 

Arrival 

None 

None 

ON 

OFF 

15 

ES3H3 

Arrival 

RCA 

None 

OFF 

ON 

16 


Arrival 

RCA 

None 

ON 

OFF 

17 


Arrival 

RCA 

RCA 

OFF 

ON 

18 

Departure 

Arrival 

RCA 

RCA 

ON 

OFF 

19 



None 

None 

OFF 

ON 

20 


Departure 

None 

None 

ON 

OFF 

21 

Departure 

Departure 

RCA 

None 

OFF 

ON 

22 

Departure 

Departure 

RCA 

None 

ON 

OFF 

23 

rmrm 

Departure 

RCA 

RCA 

OFF 

ON 

24 

Departure 


RCA 

RCA 

ON 

OFF 


Table 3. Blackbox Test Scenarios for Crossing Runway 
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m 


Traffic 


Evasive Action 
of Traffic 

Random Generator 







Traffic 

i 

Taxi 

Arrival 

None 

None 

OFF 

ON 

2 

Taxi 

Arrival 

None 

None 

ON 

OFF 

3 

Taxi 

Arrival 

RCA 

None 

OFF 

ON 

4 

Taxi 1 

Arrival 

RCA 

None 

ON 

OFF 

5 

Taxi 

Arrival 

RCA 

RCA 

OFF 

ON 

6 

Taxi 

Arrival 

RCA 

RCA 

ON 

OFF 

7 

Arrival 

Taxi 

None 

None 

OFF 

ON 

8 

Arrival 

Taxi 

None 

None 

ON 

OFF 

9 

Arrival 

Taxi 

RCA 

None 

OFF 

ON 

10 

Arrival 

Taxi 

RCA 

None 

ON 

OFF 

11 

Arrival 

Taxi 

RC* 

RCA 

OFF 

ON 

12 

Arrival 

Taxi 

RCA 

RCA 

ON 

OFF 

13 

Taxi 

Departure 

None 

None 

OFF 

ON 

14 

Taxi 

Departure 

None 

None 

ON 

OFF 

15 

Taxi 

Departure 

RCA 

None 

OFF 

ON 

16 

Taxi 


RCA 

< None 

ON 

OFF 

17 

Taxi 

Departure 

RCA 

RCA 

OFF 

ON 

18 

Taxi 

Departure 

RCA 

RCA 

ON 

OFF 

19 

Departure 

Taxi 

None 

None 

OFF 

ON 

20 

Departure 

Taxi 

None 

None 

ON 

OFF 

21 

Departure 

Taxi 

RCA 

None 

OFF 

ON 

22 

Departure 

Taxi 

RCA 

None 

ON 

OFF 

23 

Departure 

Taxi 

RCA 

RCA 

OFF 

ON 

24 



RCA 

RCA 

ON 

OFF 


Table 4. Blackbox Test Scenarios for Single Runway 
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Table 5. RIAAS, Single Runway-Comparison of Analysis Program output with 

Simulator output 


13 

































































































































Table 6. RSM, Single Runway- Comparison of Analysis Program output with 

Simulator Output 
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Table 7. RIAAS, Crossing Runway- Comparison of Analysis Program output with 

Simulator Output 
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Table 8. RSM, Crossing Runway- Comparison of Analysis Program output with 

Simulator Output 
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Table 9. Statistics of R1AAS -Single Runway, Collision Information. 


Note: The above table has an immediate implication of the case where a collision can be 
avoided. If To be the Ownship Start time and Tt be the traffic Start Time and die closest 
seperation between Ownship and traffic be Tc. The time gap between the start time of 
ownship or traffic to the time at which both ownship and traffic are closest Is Tg= Tc- 
maximum( Tt,To). As long as Tg is smaller than 5 seconds, the collision is very difficult 
to avoid, even if an alert is generated. However, if the Tg is between 10-15, a collision 
can be avoided upon noting an alert In the particular case #7 of collisions with alerts, 
there are 421 cases for which the time gap Tg is about 10-15 secs. These are all collisions 
generated with alerts with longer time gap, hence a collision can be avoided. 10-15 
seconds of time and collision can be avoided. Collision cannot be avoided in the cases for 
time gap which lies between 0-5secs. 
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Table 10. Statistics erf RSM-S ingle Runway, Collision Information 


Note: As Mentioned in the above note of table 7, if Tg is between 10-15. a collision can 
be avoided upon noting an alert In the particular case #2 of collisions with alerts, there 
are 423 cases for which the time gap Tg is about 10-15 secs. These are all collisions 
generated with alerts with longer time gap, hence a collision can be avoided.10-15 
seconds of time and collision can be avoided. Collision cannot be avoided in the cases for 
time gap which lies between 0-5secs. 
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The time gap between the start lime of ownshlp or traffic to ttw time at which both ownshlp 

and traffic are closest 



HotirToHlnlnn wH 

thout darts 

i 

IS --.111. I 

BBvOlHSIOn Wlul 896VT8 


<0 

0-5 

5- 

10 

10-15 

15-20 

20-25 

>25 

<0 

0-5 

5- 

10 

10-15 

15-20 

20-25 

>25 I 

1 

ra 

0 

kb 

0 

0 

0 

KB 

ra 

(BH 

0 

0 


0 

Bfl 

2 

ra 

0 

0 

0 

0 

0 

KB 

0 

1 

239 
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Table 11. Statistics of RSM-Single Runway, Near Collision Information 


Note: As mentioned in die above note of table 7, if the Tg is between 10-15, a collision 
can be avoided upon noting an alert In die particular case #2 of collisions with alerts, 
there are 109 cases for which the time gap Tg is about 10-15 secs. These are all collisions 
generated with alerts with longer time gap, hence a collision can be avoided. 10-15 
seconds of time and collision can be avoided. Collision cannot be avoided in die cases for 
time gap which lies between 0-5secs. 
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The time gap between the start time of ownship or traffic to the tone at which both ownship 
and traffic are closest. 


NearColllslon without alerts 


NmrCollisJon with alerts 


<0 


0-5 


5 - 

10 


10-15 


15-20 


20-25 


>25 


<0 


0-5 


5 - 

10 


10-15 


15-20 


20-25 


>25 


239 


109 


102 


339 


50 


266 


6 


9 


31 


9 


239 


109 


102 


339 


8 


9 


10 


11 


239 


109 


102 


339 


12 


13 


44 


65 


34 


14 


15 


20 


39 


57 


57 


84 


39 


114 


16 


13 


33 


44 


41 


142 


17 


20 


51 


17 


18 


19 


404 


101 


201 


40 


20 


13 


27 


69 


41 


67 


51 


21 


54 


87 


45 


49 


40 


22 


13 


15 


15 


23 


24 


13 


35 


able 12. Statistics of RIAAS -Single Runway, Near Collision Information 


Note: As mentioned in the above note of table 7, if Tg is between 10-15, a collision can 
be avoided upon noting an alert In the particular case #2 of collisions with alerts, there 
are 109 cases for which die time gap Tg is about 10-15 secs. These are all co llisions 
generated with alerts with longer time gap, hence a collision can be avoided. 10-15 
seconds of time and collision can be avoided. Collision cannot be avoided in the cases for 
time gap which lies between 0-5secs. 
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Table 13. Statistics of RIAAS-Crossing Runway, Collision Information 

Note: As mentioned in die above note of table 7, if Tg is greater than 25s, a collision can 
be avoided upon noting an alert In die particular case #22 of collisions with alerts, there 
are 167 cases for which die time gap Tg is greater than 25 secs. These are all collisions 
generated with alerts with longer time gap, hence a collision can be avoided Greater than 
25 seconds of time and collision can be avoided. Collision cannot be avoided in the cases 
for time gap which lies between 0-5secs. 
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Table 14. Statistics of RSM-Crossing Runway, Collision Information 


Note: As mentioned in die above note of table 7, if Tg is greater than 25s, a collision can 
be avoided upon noting an alert In die particular case #20 of collisions with alerts, there 
are 167 cases for which die time gap Tg is greater than 25 secs. These are all collisions 
generated with alerts with longer time gap, hence a collision can be avoided. Greater than 
25 seconds of time and collision can be avoided. Collision cannot be avoided in the cases 
for time gap which lies between 0-5secs. 
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Table 15. Statistics of RIAAS-Crossing Runway, Near Collision Information 
Note: There is no info rmatio n related to the above mentioned note of table 9. 
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Tablel6. Statistics of RSM-Crossing Runway, Near Collision Information 


Note: There is no information related to die above mentioned note of table 9. 
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5. Conclusion 


In this report, we have s ummar ized the results that we obtained in running the RIAAS 
and RSM software and attempted to evaluate their efficacy in avoiding collision/near- 
collision situations. In this process, we have categorized different runs into four primary 
classes depending the outcome. The information provided here should be used to further 
improve RIAAS and RSM software. 
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